273109 - Japanese attachment file name garbled, after fix for Bug 243504 and Bug 263462

Status: RESOLVED DUPLICATE of bug 274264

(Thunderbird :: Message Compose Window, defect)

Description

[Ahn Dal-Soo]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; ja-JP; rv:1.7.5) Gecko/20041108 Firefox/1.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; ja-JP; rv:1.7.5) Gecko/20041108 Firefox/1.0

Symptom: Japanese attachment file name gets garbled in the following condition.

Condition:
- Japanese character set used in file system is ShiftJIS (i.e Windows environment).
- The 2nd byte of ShiftJIS character is less than 0x80 (i.e range that overlaps
with ASCII). ShiftJIS characters may contain 0x40-0x7E and 0x80-0xFC in 2nd byte.
- The 2nd byte of ShiftJIS character is escaped. 0x5C is the case.

Related code:
/intl/uconv/src/nsUTF8ConverterService.cpp

nsUTF8ConverterService::ConvertURISpecToUTF8(const nsACString &aSpec, const char
*aCharset, nsACString &aUTF8Spec)
{
-snip-
PRBool written = NS_UnescapeURL(PromiseFlatCString(aSpec).get(), aSpec.Length(),
esc_OnlyNonASCII, unescapedSpec);
-snip-
return ToUTF8(unescapedSpec, aCharset, aUTF8Spec);
}

What is going on:
1. For instance, the URI of attachment file name is
"file:///C:/Documents%20and%20Settings/me/%83f%83X%83N%83g%83b%83v/%83%5C.txt
"
2. Calls AttachmentPrettyName().
3. Calls NS_UnescapeURL() with esc_OnlyNonASCII flag.
4. Does not unescape 0x5C.
5. Calls ToUTF8(). The 2nd byte of ShiftJIS character is still escaped.
6. "5C" shows up in mail composition window. Refer to the attached screenshot.

Reproducible: Always
Steps to Reproduce:
1. Compose a new message.
2. Attach a file with Japanese name that contains some particlar character codes.
3. Displayed attachement file name is garbled.
Actual Results:  
Raw character code shows up like "5C".

Expected Results:  
Readable Japanese characters show up.

Comment 1

[Ahn Dal-Soo]

Attachment: Screenshot

Screenshot of garbled attachement file name.

Comment 2

[u69748]

I can reproduce this bug.
Nominating blocking 1.0

Comment 3

[Koike Kazuhiko]

This bug should be fixed ASAP. Many people make file with Japanese name
and attach it. Please reconsider blocking 1.0.

Comment 4

[Scott MacGregor]

is this a new problem or something that's always been there?

Comment 5

[u69748]

I think this is a 0.9 regression bug.

Comment 6

[Scott MacGregor]

jshin can you take a look at this? I don't see any changes in our codebase to
our UTF8 converter since 2003 so I don't understand how this is a recent
regression unless the change is with our escaping code. 

Possible sources could be (although after looking back at that patch, I doubt it)

https://bugzilla.mozilla.org/show_bug.cgi?id=268219

Comment 7

[Ahn Dal-Soo]

This is new since 0.9. I checked http://bugzilla.mozilla.gr.jp/ that is a local
bugzilla in Japan. They report this issue on 0.9 and 1.0RC1. It appears 0.7.3
works fine. There is no report regarding 0.8. NS_UnescapeURL() is a bit
different between these versions as follows.

0.7.3: ignores esc_OnlyNonASCII flag and unescapes the 2nd byte of a ShiftJIS
character.
0.8: I haven't reviewed code.
1.0RC1 and probably 0.9: handles esc_OnlyNonASCII flag and doesn't unescape the
2nd byte of a ShiftJIS character.

Comment 8

[Scott MacGregor]

Ahn,
this bug looks like the culprit:

https://bugzilla.mozilla.org/show_bug.cgi?id=243504

That looks pretty suspicious to me. What do you think?

Comment 9

[Scott MacGregor]

Sorry I didn't mean that bug. I meant this one:

https://bugzilla.mozilla.org/show_bug.cgi?id=263462

which was a I18n change to our unescaping code. I'm pretty sure this is the
culprit. Thoughts?

Comment 10

[Scott MacGregor]

do any of you have a debug build where you can quickly try backing out: 

https://bugzilla.mozilla.org/show_bug.cgi?id=263462

and confirming that this starts working again? 

Comment 11

[Scott MacGregor]

alternatively, can someone send me an e-mail with an attachment that has a name
in japanese that contains these particular characters that get corrupted?

I have japanese fonts on my machine, so if you send it to me, I can drag it to
my compose window and hopefully see the corruption. 

Comment 12

[Ahn Dal-Soo]

Scott, I sent you an email with attachment named in Japanese via web-based Yahoo
mail. Then I noticed that Firefox 1.0 too causes character corruption when I
save attachement with Japanese name to my local disk. At ordinay web site, it is
rare to download files with Japanese name. But at web-based email site, we often
find Japanese attachment name. I will report this unless it is already reported.

Comment 13

[Scott MacGregor]

I got the e-mail attachment just fine. My problem is that on Windows XP when I
save the file, windows converts it into my native charset so the file on my OS
looks like __.txt. I'm trying to figure out a way to get my Operating system to
save it without losing the japanese characters. 

Comment 14

[Scott MacGregor]

ok, in about 15 minutes, a test build will be up on the ftp servers:

ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/test/ThunderbirdSetup.exe

This test build backs out the patch for Bug #263462. 

Can one of you guys try it out and see if that fixes this issue as soon as you
can? If we can confirm that it does we'll look into re-spinning our 1.0 bits
with the back out. Thanks!

Comment 15

[u69748]

Mscott, it works fine!

Comment 16

[Ahn Dal-Soo]

Works fine on my environemnt too. Thank you!
The issue I mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=273109#c12
appears to be a duplicate of https://bugzilla.mozilla.org/show_bug.cgi?id=252309

Comment 17

[Scott MacGregor]

we are respinning the final 1.0 bits to pick up the backout cited above. Marking
fixed-aviary1.0.

Leaving open and re-assigning to jshin so he can fix his original change in Bug
#263462 to not break on certain japanese characters.

Comment 18

[WADA:World Anti-bad-Duping Agency]

Patch for Bug 263462 is backed out on Thunderbird 1.0 release build.
This probably revive Bug 243504 and Bug 239192 (problem in file attach when "#"
or ";" in file path).
Both bugs seems to be resolved by patch for Bug 243504 in conjunction with patch
for Bug 263462 (Bug 243504 depends on Bug 263462). 
See post in MozillaZine forum. 
 ( http://forums.mozillazine.org/viewtopic.php?p=1048564#1048564 )

Comment 19

[WADA:World Anti-bad-Duping Agency]

Change summary(add description on Bug 243504 and Bug 263462 according to Comment
14 by Scott).

Comment 20

[Ahn Dal-Soo]

Attachment: Patch to leave ascii symbols escaped

Comment 21

[Ahn Dal-Soo]

Attached is a patch to leave ascii symbols like # escaped. Applying both of this
and jshin's patch resolves all of Bug 239192, Bug 243504, Bug 263462 and
Bug 273109.

Comment 22

[Ahn Dal-Soo]

Perhaps esc_FilePath looks better than the combination of esc_Scheme | esc_Host
| esc_Directory | esc_FileBaseName | esc_FileExtension. I just didn't find 
documentation about it in nsEscape.cpp.

Comment 23

[Hidehiro Kozawa]

Do Mozilla Suite(Trunk and 1.7branch) have same problem?
Patch of Bug 263462 was check in for them too. but not backout.

Comment 24

[WADA:World Anti-bad-Duping Agency]

Add to list in Bug 274366 (regressions meta-bug)

Comment 25

[Jungshik Shin]

Comment on attachment 168486 [details] [diff] [review]
Patch to leave ascii symbols escaped

Yes, this should , along with my patch, fix all those things. IDN host name
could be a problem, though. We also have to be careful as to which set of flags
to use.

Comment 26

[Jungshik Shin]

This is fixed on trunk (bug 274264) and the patch will be fixed in  branches
asas I get approved.
I'm resolving it as a dupe of bug 274264 (because there's a landed patch). No
need to keep two bugs around. 


*** This bug has been marked as a duplicate of 274264 ***

Comment 27

[WADA:World Anti-bad-Duping Agency]

This bug is for Thunderbird, and was closed as DUP of bug 274264 which is for
Mozilla App Suite.
But bug 274264 was reopened on 2005/1/21 due to backout of fix for it.
So reopen this bug in order to request blocking1.8rc1 (Thunderbird 1.5 rc1).

Comment 28

[WADA:World Anti-bad-Duping Agency]

This bug's problem have revived on Tb 1.5 Beta2. (See bug 274264 comment #27)
This is due to backout of patch for bug 274264 from trunk (See bug 274264
comment #26) 

Comment 29

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(away 4/27 - 5/6)]

This is not a bug of Tb. It's Core bug.

*** This bug has been marked as a duplicate of 274264 ***